<!doctype html>
<html lang="zh-cn">
<head>

    <meta charset="utf-8">
    <meta name="generator" content="Hugo 0.57.2" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <title>Elk | The Sky of OtsWang</title>
    <meta property="og:title" content="Elk - The Sky of OtsWang">
    <meta property="og:type" content="article">
        
    <meta property="article:published_time" content="2019-05-30T17:09:25&#43;08:00">
        
        
    <meta property="article:modified_time" content="2019-05-30T17:09:25&#43;08:00">
        
    <meta name="Keywords" content="golang,go语言,otswang,java,博客,python">
    <meta name="description" content="Elk">
        
    <meta name="author" content="OtsWang">
    <meta property="og:url" content="https://otswang.gitee.io/hugo/post/middle_tools/elk/">
    <link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">

    <link rel="stylesheet" href="/hugo/css/normalize.css">
    
        <link rel="stylesheet" href="/hugo/css/prism.css">
    
    <link rel="stylesheet" href="/hugo/css/style.css">
    <script type="text/javascript" src="//cdn.bootcss.com/jquery/3.2.1/jquery.min.js"></script>

    


    
    
</head>

<body>
<header id="header" class="clearfix">
    <div class="container">
        <div class="col-group">
            <div class="site-name ">
                
                    <a id="logo" href="https://otswang.gitee.io/hugo/">
                        The Sky of OtsWang
                    </a>
                
                <p class="description">擅长写HelloWorld的小小码农</p>
            </div>
            <div>
                <nav id="nav-menu" class="clearfix">
                    
                    
                    <a  href="https://otswang.gitee.io/hugo/" title="Home">Home</a>
                    
                    <a  href="https://otswang.gitee.io/hugo/tags/" title="Tags">Tags</a>
                    
                    <a  href="https://otswang.gitee.io/hugo/categories/" title="Categories">Categories</a>
                    
                    <a  href="https://otswang.gitee.io/hugo/archives/" title="Archives">Archives</a>
                    
                    <a  href="https://otswang.gitee.io/hugo/about/" title="About">About</a>
                    
                </nav>
            </div>
        </div>
    </div>
</header>


<div id="body">
    <div class="container">
        <div class="col-group">

            <div class="col-8" id="main">
                <div class="res-cons">
                    <article class="post">
                        <header>
                            <h1 class="post-title">Elk</h1>
                        </header>
                        <date class="post-meta meta-date">
                            2019年5月30日
                        </date>
                        
                        <div class="post-meta">
                            <span>|</span>
                            
                                <span class="meta-category"><a href="https://otswang.gitee.io/hugo/categories/middletools">MiddleTools</a></span>
                            
                        </div>
                        
                        
                        
                        <div class="post-content">
                            <p>elk 日志系统</p>

<p>完整的日志数据具有的作用：</p>

<ul>
<li>信息查找： 检索日志消息，定位dub，找出解决方案</li>
<li>服务诊断： 通过日志信息统计分析服务器的负荷和运行状态</li>
<li>数据分析： 格式话的log，可以做数据分析，聚合统计出有意义的信息。</li>
</ul>

<p>Elasticsearch： 基于Lucene的开源分布式搜索服务器。</p>

<p>Logstash： 对日志收集，过滤，分析，并将数据存储供以后使用。</p>

<p>Kibana： 前端展示工具。</p>

<p>一个elk架构图如下：</p>

<p><img src="/hugo/src/img/middle/elk-normal.png" /></p>

<h2 id="elasticsearch">Elasticsearch</h2>

<h2 id="logstash">Logstash</h2>

<p>input:</p>

<ul>
<li>file 从文件中读取，类似于 tailf</li>
<li>syslog 在514端口监听系统日志信息</li>
<li>redis 从redis server中读取数据</li>
<li>beats 从fileBeats中读取数据</li>
</ul>

<p>Filters:</p>

<ul>
<li>grok 解析文本数据</li>
<li>mutate 字段进行转换</li>
<li>drop 丢弃</li>
<li>clone 拷贝event</li>
<li>geoip 添加地理信息</li>
</ul>

<p>Codec：</p>

<p>基于流的过滤器</p>

<ul>
<li>JSon 使用JSon对数据解码与编码</li>
<li>multiline 将多个事件中的数据汇总为一个单一的行，比如java的堆栈异常信息</li>
</ul>

<p>input 的 file  模式：</p>

<p>type：指定一个名称，设置type后，可以在后面的filter和output中对不同的type做不同的处理，适用于需要消费多个日志文件的场景。</p>

<p>sincedb_path：通常Logstash会记录每个文件已经被读取到的位置，保存在sincedb中，如果Logstash重启，那么对于同一个文件，会继续从上次记录的位置开始读取。如果想重新从头读取文件，需要删除sincedb文件，sincedb_path则是指定了该文件的路径。为了方便，我们可以根据需要将其设置为“/dev/null”，即不保存位置信息。</p>

<pre><code class="language-conf">input {
    file {
        type =&gt; &quot;example_nginx_access&quot;
        path =&gt; [&quot;/var/log/nginx/example.access.log&quot;]
        start_position =&gt; &quot;beginning&quot;
        sincedb_path =&gt; &quot;/dev/null&quot;
    }
}
</code></pre>

<p>output 的 redis 模式;</p>

<pre><code class="language-conf">output {
    if [type] == &quot;example_nginx_access&quot; {
        redis {
            host =&gt; &quot;127.0.0.1&quot;
            port =&gt; &quot;6379&quot;
            data_type =&gt; &quot;list&quot;
            key =&gt; &quot;logstash:example_nginx_access&quot;
        }
      #  stdout {codec =&gt; rubydebug}
    }
}
</code></pre>

<p>转接的配置</p>

<pre><code class="language-conf"># 从redis输入数据
input {
    redis {
            host =&gt; &quot;127.0.0.1&quot;
            port =&gt; &quot;6379&quot;
            data_type =&gt; &quot;list&quot;
            key =&gt; &quot;logstash:example_nginx_access&quot;
    }
}
 
output {
    elasticsearch {
        index =&gt; &quot;logstash-example-nginx-%{+YYYY.MM}&quot;
        hosts =&gt; [&quot;127.0.0.1:9200&quot;]
    }
}
</code></pre>

<p>filter 的知识：</p>

<p>待添加</p>

<h2 id="kibana">Kibana</h2>

<p>参考资料：</p>

<p><a href="https://www.cnblogs.com/kevingrace/p/5919021.html">https://www.cnblogs.com/kevingrace/p/5919021.html</a></p>
                        </div>

                        


                        


                        <div class="post-meta meta-tags">
                            
                            <ul class="clearfix">
                                
                                <li><a href="https://otswang.gitee.io/hugo/tags/elk">ELK</a></li>
                                
                            </ul>
                            
                        </div>
                    </article>
                    
    

    
    
                </div>
            </div>
            <div id="secondary">

    <section class="widget">
        <form id="search" action="//www.google.com/search" method="get" accept-charset="utf-8" target="_blank" _lpchecked="1">
      
      <input type="text" name="q" maxlength="20" placeholder="Search">
      <input type="hidden" name="sitesearch" value="https://otswang.gitee.io/hugo/">
      <button type="submit" class="submit icon-search"></button>
</form>
    </section>

    
    <div class="clear">
        <div class="toc-article">
            <div class="toc-title">文章目录</dixsv>
            <nav id="TableOfContents">
<ul>
<li>
<ul>
<li><a href="#elasticsearch">Elasticsearch</a></li>
<li><a href="#logstash">Logstash</a></li>
<li><a href="#kibana">Kibana</a></li>
</ul></li>
</ul>
</nav>
        </div>
    </div>
    

</div>
        </div>
    </div>
</div>
<footer id="footer">
    <div class="container">
        &copy; 2020 <a href="https://otswang.gitee.io/hugo/">The Sky of OtsWang By OtsWang</a>.
        Powered by <a rel="nofollow noreferer noopener" href="https://gohugo.io" target="_blank">Hugo</a>.
        <a href="https://www.flysnow.org/" target="_blank">Theme</a> based on <a href="https://github.com/Dudiao137/maupassant-hugo" target="_blank">maupassant-ots</a>.
        
    </div>
</footer>


    <script type="text/javascript">
    
    (function(){
        $("pre code").parent().addClass("line-numbers")
    }())

    window.MathJax = {
        tex2jax: {
            inlineMath: [ ['$','$'] ],
            processEscapes: true
        }
    };
    </script>
    <script type="text/javascript" src="/hugo/js/prism.js" async="true"></script>
    <script src='https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-MML-AM_CHTML' async></script>

<a id="rocket" href="#top"></a>
<script type="text/javascript" src="/hugo/js/totop.js?v=0.0.0" async=""></script>







 
 <script src="https://mermaidjs.github.io/scripts/mermaid.min.js"></script>
 <script>
       mermaid.initialize({ startOnLoad: true });
 </script>
</body>
</html>
